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Objectives: The French National Occupational Diseases Surveillance and Prevention Network (RNV3P) is a French network of oc- 
cupational disease specialists, which collects, in standardised coded reports, all cases where a physician of any specialty, referred 
a patient to a university occupational disease centre, to establish the relation between the disease observed and occupational ex- 
posures, independently of statutory considerations related to compensation. The objective is to compare the relevance of dispro- 
portionality measures, widely used in pharmacovigilance, for the detection of potentially new disease x exposure associations in 
RNV3P database (by analogy with the detection of potentially new health event x drug associations in the spontaneous reporting 
databases from pharmacovigilance). 

Methods: 2001-2009 data from RNV3P are used (81,132 observations leading to 11,627 disease x exposure associations). The 
structure of RNV3P database is compared with the ones of pharmacovigilance databases. Seven disproportionality metrics are 
tested and their results, notably in terms of ranking the disease x exposure associations, are compared. 

Results: RNV3P and pharmacovigilance databases showed similar structure. Frequentist methods (proportional reporting ratio 
[PRR], reporting odds ratio [ROR]) and a Bayesian one (known as BCPNN for "Bayesian Confidence Propagation Neural Network") 
show a rather similar behaviour on our data, conversely to other methods (as Poisson). Finally the PRR method was chosen, be- 
cause more complex methods did not show a greater value with the RNV3P data. Accordingly, a procedure for detecting signals 
with PRR method, automatic triage for exclusion of associations already known, and then investigating these signals is suggested. 
Conclusion: This procedure may be seen as a first step of hypothesis generation before launching epidemiological and/or experi- 
mental studies. 
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Newly Identified Health Risks (SCENHIR) stated recently 
"There are a number of reasons why an emerging issue was 
not identified at an appropriate time or its potential efi^ects 
were not properly considered." [1]. The two first - among the 8 
"reasons of past failure" identified - are "Inadequate monitor- 
ing/surveillance resulting in a failure to detect the presence of 
a disease and/or agent at an early stage" and "Failure to make 
important relevant information available to the risk assessors/ 
risk managers". 

In the occupational health field, occupational registers 
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from insurances systems, because of their primary vocation of 
compensation, are devoted to already known "disease x expo- 
sure (D X E)" associations and are not supposed to report and 
detect new D x E associations. That's the reason why occupa- 
tional diseases (OD) surveillance schemes have been set out by 
university physicians and researchers specialists of OD, with 
the ambition to take part efficiently to the occupational health 
surveillance. Some have developed a systematic collection of 
all OD cases by the mean of large samples of physicians from 
different specialties reporting continuously or periodically 
(surveillance action), as the Health and Occupation Research 
Network system does in UK [2-5] . Some rely on a highly quah- 
tative approach consisting of the analysis by OD specialists 
(expert function) of the work attributability (imputabUity) of 
disease cases whose work-relatedness has raised questions for 
physicians (vigilance action). 

"The French National Occupational Diseases Surveil- 
lance and Prevention Network (RNV3P)" is a French network 
of academic specialists of ODs, which belongs to the last 
category. Indeed, RNV3P collects since 2001 in its database, 
in standardised coded reports, all cases where a physician of 
any specialty, and throughout metropolitan France, referred 
a patient to a university OD centre, to establish the relation 
between the disease observed and one or several occupational 
exposures, independently of statutory considerations related to 
compensation [6-8]. 

One of the research aims of RNV3P is to improve its 
ability for early detection of new ODs, especially find out in- 
formation that could be useful to achieve this objective within 
its growing database. In order to achieve this aim, methods 
for detecting emerging diseases that are currently used in oc- 
cupational health and by vigilance systems in other healthcare 
fields were first analysed [9] . Most of the data mining methods 
used in health surveillance (especially syndromic surveillance) 
aimed at identifying unusual increases in the number of cases 
within chronological data [10-12]. Beside these methods, we 
identified the "disproportionality metrics" used in pharma- 
covigilance [13,14] as most relevant to our aim of revealing 
currently unknown or poorly-documented D x E associations 
("Disease" might stand for "health effect" or "health event", 
and "exposure" for "drug" in the pharmacovigUance context). 
Some disproportionality metrics are frequency-based meth- 
ods (chi^, proportional reporting ratio [PRR] which is used in 
the British pharmacovigilance database, reporting odds ratio 
[ROR], Yules' Q), some are based on Poisson's law (Poisson 
and Sequential Probability Ratio Test [SPRT2]), and some rely 
on Bayesian methods (as Bayesian Confidence Propagation 
Neural Network [BCPNN], which is used in the international 
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pharmacovigilance database). 

In pharmacovigilance, these methods are dedicated to 
"hypothesis generation" , also called " signal generation" , where sig- 
nal stand for "reported information on a possible relationship 
between an adverse event and a drug, of which the relationship 
is unknown or incompletely documented previously" (World 
Health Organization [WHO] definition). These "Safety Data 
Mining" methods have shovm high potentiailities in the analysis 
of the very large pharmacovigilance spontaneous reporting 
databases (e.g., more than 250,000 reports annually for the 
Uppsala Monitoring Centre [UMC] "WHO" database). Refro- 
spective studies have shown that the utilisation of these meth- 
ods 1) confirmed signals that had been clinically first identified, 
2) might highhght these new associations sooner, 3) might dis- 
tinguish a specific adverse drug effect of a molecule, not shared 
by its whole therapeutic family [15]. Conversely we may also 
remark that they are few examples of new drug x health effect 
that have been first highlighted by these methods prospectively, 
probably because, these methods have not been used for such a 
long time, but mainly because the analysis of numerous signals 
is also time consuming for experts. 

We first tested the PRR method on a RNV3P sample few 
years ago [16], and the interest of PRR in the RNV3P was also 
highlighted in a second pubUcation with twice more data and 
the example of systemic sclerosis [8] . 

The objectives of this paper are to present the compari- 
son of the seven previously cited pharmacovigilance methods 
tested on RNV3P, in order to choose the most promising 
method(s), and then propose a procedure for an utilisation of 
these method(s) on a routine basis in the RNV3P network. 

Materials and Methods 

Data 

RNV3P data from 2001-2009 were analysed, including a total 
of 81,132 observations. The main codes used are International 
Classification of Diseases 10th Revision (ICD-10), Internation- 
al Standard Classification of Occupations (ISCO 88), Nomen- 
clature d'Activites Fran? aises for activity sector (NAF 93), and 
a French code for exposures from the French national insur- 
ance system of salaried workers (CNAM-TS), at the origin of 
the European Occupational Diseases Statistics Classification of 
the causal agents of the ODs (EODS causal agents code). This 
last code may be downloaded on the following EU website: 
http://circa.europa.eu/Public/irc/dsis/hasaw/library?l=/oc- 
cupational_statisstics/working_paper_l 8/_EN_l .0_ 

One key point of RNV3P data is that 75% of the observa- 
tions notify only one exposure (17% with 2 exposures, 5% with 
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3, 2% with 4, < 1% with 5 exposures notified). 
Methods 

Pharmacovigilance methods are detailed in several publica- 
tions, where especially formula and signal generation criteria 
may be found [17,18]. At the first level, each report generates n 
"D X E associations", « being the number of exposures notified 
in this report notifying one disease. At the aggregated level (the 
whole database being taken into account), the observed number 
of cases of each "D x E association" is compared with the esti- 
mation of the expected number of cases. This expected number 
of cases is calculated using data from the database only, reason 
why these methods are also named "numerator dependent" . 
Pharmacovigilance methods generate a statistical signal when 
there is a discrepancy between the observed number of a D x 
E association within the database and the expected number of 
cases. Several methods are proposed to estimate the magnitude 
of the disproportion measure and its confidence interval. AH 
are derived from a basic 2x2 contingence table generated for 
each D X E pair [16]. In other words, these methods are using 
the "background noise" summarized in the marginal counts of 
these 2x2 contingency tables. 

Seven disproportionality metrics were applied and their 
results compared: the usual, frequency -based methods (n°l: 
PRR which is used in the British pharmacovigilance database, 
n°2: ROR, n°3: Yules, n°4: chi^), methods based on Poisson's 
law (n°5: Poisson and n°6 : SPRT2), as well as one of the 
two Bayesian methods that have been described to date (n°7: 
BCPNN), which is used by the WHO collaborative centre on 
pharmacovigilance (UMC) on the international pharmaco- 
vigilance database. Only the Empirical Bayes Screening (EBS) 
method, also named "Dumouchel method", was not tested, as 
it was more complex to implement. 

Another point that should be kept in mind is that the same 
method may be used with different signal generation criteria. 
For instance there are two common PRR signal generation 
criteria for PRR method. The first one is a composite criterion 
requiring that the number of observed cases (a) is at least equal 
to 3, and that PRR and chi^ measures for this association are at 
least equal to 2 and 4 respectively: a > 3 and PRR > 2 and chi^ 
> 4 (named "PRR," in Table 1). The second is that the lower 
bound of its 95% confidence interval has to exceeded one: LIjj^ 
(PRR) > 1 (named "PRRj" in Table 1). 

The comparison of the methods can't be done with sensi- 
tivity and specificity assessment, due to the absence of a Gold 
Standard allowing differentiating true positives (health events 
with causaUty relation to the notified exposure) from false posi- 
tives (see Discussion). For that reason, we propose a compari- 



son of the behaviours of the methods, notably of the ranking 
of the disproportionality measure affected to the D x E asso- 
ciations. 

Software 

All analyses are conducted with SPlus 6.1 (Insightful Co., Se- 
atde, WA, USA). 

Results 

structure of the D x E database 

The 81,132 observations lead to 11,627 "D x E" associations. 
The filling percentage of the theoretical "D x E" matrix is of 
1.2%, which is not so different from pharmacovigilance data- 
bases (e.g., 2.1% for the US Medwatch database in 2002 [19]), 
or 2.2% to 3.6% for the WHO Vigibase in 2005, according to 
the level of information (code precision) taken into account 
(personal communication with UMC). Most of the cases are 
reported once (63% of the associations). Four thousands and 
two hundreds ninety one D x E associations were reported 
more than twice and are potentially candidate for generating 
a signal. The distribution of the D x E associations, according 
to their number of observations (Fig. 1), is similar to the one 
of the French pharmacovigilance database. This shows a good 
comparability of RNV3P D x E matrix and the health event x 
drug matrix of the pharmacovigilance databases, according to 
their sparsity ("empty matrixes") and to the distribution of the 
size of theses associations. The main difference is that RVN3P 
database is buUt on a smaller number of cases. The distribution 
is also similar for the RNV3P "disease x occupation" and "dis- 
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Fig. 1. "Disease x exposure (D x E)" associations according to the 
number of times they have been notified in the French National 
Occupational Diseases Surveillance and Prevention Network (i.e., 
number of similar observations or reports). 
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ease x activity sector" matrices, that were also studied (results 
not shown). 



Comparison of the number of signals generated by 
the different methods 

Table 1 shows the number of signals generated by the 7 meth- 
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Fig. 2. Comparison of the behaviour of the proportional reporting ratio (PRR) method (x axis), with the disproportionality metrics reporting 
odds ratio (ROR), Yules, chP, Poisson, Sequential Probability Ratio Test (SPRT2), Bayesian Confidence Propagation Neural Network (BCPNN), 
according to the number of reports in each disease x exposure associations (symbols). The associations represented near the origin of the axes 
have the lowest disproportionality measures, whereas the ones to the opposite have the highest measures and present the strongest signals. 
When associations are plotted near the bisecting line, a similar rank has been affected by the 2 disproportionality metrics compared. Conversely, 
when some associations are plotted lower (respectively higher) than the bisecting line, it means that they have been affected lower (respectively 
higher) disproportionality measures by the method represented on the y axis, than by the PRR method. 
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ods with their usual signal generation criterion, and if these 
signals concern potentially compensable diseases or not (be- 
cause the potentially new OD belongs to the second category, 
the signals belonging to the first category wiU not have to be 
investigated). In terms of number of signal generated, we 
can distinguish two groups of methods. PRR2, ROR, Yules, 
BCPNN and chi^ generate a signal for 21 to 23% of the D x E 
associations, and only the 12-14% not eligible for compensation 
would deserve investigation. This percentage rises up to 58% 
to 66%, if we consider only the associations reported twice or 
more. PRRi and SPRT2 are more "restrictive", as respectively 
1 1% and 8% of all associations generate a signal (6% and 3% 
of associations not eligible for compensation would deserve 
investigation). Poisson method with its common signal genera- 
tion criteria generates a signal for up to 25% of all associations. 
Table 1 also presents a quantitative assessment of the overlap 
of the signals generated by each method with the signals gener- 
ated by PRRi and PRRj taken as references. 

Comparison of disproportionality metrics with 
regards to their respective ranking of all D x E 
associations 

The differences between PRRj and the other methods might be 
understood when comparing the respective ranking of the D x 
E association by the different methods. This analysis shows 
that the behaviour of the 7 metrics is different according to the 
number of cases reported for each D x E association. This is 
illustrated by Fig. 2, which displays the rank of each D x E as- 
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sociation according to PRR2, as a reference (x axis), compared 
to the rank of this association with other methods successively 
(y axis), highlighting in the same time (symbols) the number of 
reports for each D x E association. This figure shows the slight 
similar behaviours of PRR2 with the other firequentist method 
ROR, and its derivative Yules (which rank similarly the asso- 
ciations). We also notice only a very slight difference between 
PRR2 and BCPNN: BCPNN may give a slightly superior rank 
to associations with a low number of associations, whereas the 
"frequentist" method PRR2 is more "sensitive" to the highest 
number of reports. To the contrary, there are big differences in 
terms of ranking when considering PRR in one hand, and chi^, 
SPRT2 or Poisson to the other For example, chi^ ranking dis- 
tribution is biphasic with regards to PRR2, as it is very sensitive 
to D X E associations reported a high number of times, and at 
the same time sensitive to associations reported a low number 
of times, when this number is equal or nearly equcil to the mar- 
ginal count of the exposure or of the disease. 

Finally, the relatively similar behaviour of PRR2 ^nd 
BCPNN, despite their different theoretical backgrounds, is il- 
lustrated with the example of the systemic sclerosis (ICD-10: 
M34). One hundred seventy eight observations reported M34 
as a main Diagnosis on 2001-2009 data. These data lead to 70 
associations "M34 x exposure", of which 27 associations are 
reported more than twice. The ranking of the BCPNN mea- 
sures of these "M34 x E associations", and the overlap with 
the signals generated by PRR, and PRR2 are presented (Fig. 
3). Difference in signal generation between BCPNN and PRRj 
only concerns one association (which is just exceeding the 
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BCPNN signal threshold), and differences with PRRi only con- 
cern the associations reported only twice. Furthermore, their 
ranking of the highest signals is similar (Fig. 4). 

Discussion 

In this work, 7 disproportionality metrics used in pharmaco- 
vigUance to highlight potential new "drugs x health events" 
associations were tested on RNV3P database to assess their 
relevance for detecting new "exposure x health events". Their 
results, in terms of number of signals generated and signal 
ranking were compared, and an illustration was given with 
the disease Systemic Scleroderma. This was a first step before 
choosing the most promising method(s), and proposing a pro- 
cedure for an utilisation of these method(s) on a routine basis 
in the RNV3P network. 

Choice of a method in the light of the comparisons 
made between these disproportionality metrics in 
this work and in the literature 

Considering the above mentioned comparisons, and many 
examples accurately studied (notably the temporal trend of the 
signal), the PRR method was finally chosen for integration in 
a wider procedure of screening of RNV3P database, because 
more complex methods did not show a greater value with the 
RNV3P data. As discussed in the literature, the price of sophis- 
tication is these methods is the "increased cost of lack of transpar- 
ency" [15], which seems so far not justified in our context. 

There are few data in the literature about comparison of 
these methods, because, in the absence of a Gold Standard al- 
lowing differentiating true positives (health events with causal- 
ity relation) from false positives, it is difficult to calculate their 
respective sensitivity and specificity. That's why, the estimations 
of sensitivity and specificity often rely on simulated data [17]. 
When considering all the D x E association generating a signal 
with their simulated data, the authors did not show many dif- 
ferences in specificity according to the methods, and slight dif- 
ferences in terms of sensitivity. When taking into account the 
ranking of all signals, they showed a less rate of false positives 
of BCPNN and EES, but this superiority to irequentist meth- 
ods disappeared soon when increasing the number of the most 
highly ranked combination taken into account, and when the 
number of observed cases is increasing (no more differences 
for associations reported at least 5 times). Finally they showed 
that all methods had a better performance, when ranking was 
done on the lower bound of their confidence interval that takes 
into account the variance of the disproportionality measure. 
We should nevertheless remember that these estimations of 



sensitivity and specificity remain relatives to the structure of 
the dataset they were tested on. That's the reason why, on other 
studies, PRR may have shown superiority in some contexts [20]. 
As Hauben and Bate [15] said, "Judicious implementation of 
all the methods gives comparable results and far greater varia- 
tion in performance is seen owing to heterogeneity in imple- 
mentation choices, such as threshold election/titration and the 
triage logic and procedures for investigation of signails". In the 
light of this comment, we could analyse the added value, for our 
purpose of some sophistications as the weighting of some D x 
E associations to allow them generate a signal sooner if they 
appear (e.g., if there has been suspicion due to experimental 
toxicological data. Quantitative Structure Activity Relation- 
ships [QSARs], or even clinical sentinel approach). Conversely, 
stratification according to age, sex or other variables is not yet 
relevant, as the number of cases per associations is already low 
(it would prevent many D x E associations to reach statistical 
significance). 

Proposed procedure for the routine use in RNV3P 

Accordingly, a procedure for detecting signals with PRR meth- 
od, and then investigating these signals has been suggested: 
1) detection of signals using the PRR method; 2) automatic 
sorting and elimination of signals generated that can lead to 
recognition of an OD as they reflect known associations; 3) 
investigation of each as yet unknown D x E association gener- 
ating a signal, by investigating data within the database (number 
and source of cases, changes over time of the signal, distribu- 
tion of the attributabiUties assessed by the OD specialists, etc); 4) 
analysis of data in the literature using an algorithm explaining 
the level of proof from, on one hand human data (epidemiol- 
ogy and case reports), and to the other hand experimental data 
(toxicology), and using Bradford-Hill causality criteria. This 
procedure should be used in the next future on a routine ba- 
sis, as soon as the new RNV3P information system-currently 
tested-wUl allow welcoming a complementary analysis module. 

Interest and limits of this procedure 

We think this procedure including disproportionality metrics 
represent an interesting supplemental quantitative tool that can 
be helpful to highlight some potentially emerging D x E asso- 
ciations which should require attention and specific investiga- 
tions (cf examples listed in the previous publications [8, 16]). 

Yet, it is important to remember that these methods are 
only the first step in a more comprehensive process, which 
requires evaluation of the relevance of the signals generated, 
and monitoring of such signals ("signal strengthening", "signal 
foUow-up"). These methods are not able to demonstiate causal- 
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ity. They may be seen as a first step of hypothesis generation 
before launching epidemiological and/or experimental studies. 
For that reason, if these methods free us from time to screen 
"manually" the RNV3P database in the search of new associa- 
tions not visible from usual statistic analyses, in turn they re- 
quire from us much time to analyse the signals generated, and 
exclude the obvious false positive. Another limit is that these 
methods do not take into account the effects of multi-exposure. 
That's a reason why, we also develop methods that will take 
into account this important point [21]. 

Finally, we should not forget that to discover potentially 
new D X E associations, the first and most important point is 
that our scheme continuously increases its ability to capture 
those cases (i.e., that our "vigie/vigilance" action is well-known 
from a majority of physicians who would refer to us these cas- 
es), and that our coding allow information to be of high quality 
and accurate (dynamic evolution of the codes, homogeneity of 
coding practices, etc). 
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